Driver Drowsiness Monitoring System using Machine Learning

Abstract

In today\'s world, sleepiness is one of the main causes of road accidents, many of which have tragic outcomes. Statistics show that the majority of traffic collisions, which frequently result in fatalities and serious injuries, are caused by sleepy driving. As a result, various studies have been done to develop software that can recognize driver tiredness and alert them before making a major error. Using methods from the automobile industry, several of the more popular ways to design their own systems. However, other factors, such as vehicle type, road design, and the capacity to operate the driver\'s wheel, significantly impacted these traditional criteria. In order to monitor a driver\'s drowsiness, certain techniques use psychological methods, which frequently produce the most accurate and dependable results. These methods are expensive, though, because electrodes must be applied to the head and torso. This paper describes a machine learning approach to sleepiness detection. The areas of the driver\'s eyes are located using face detection, and these regions serve as templates for eye tracking in subsequent frames. Finally, drowsiness detection is performed on the tracked eye images to produce alarm warnings. The three steps of this method are Face detection, Eye detection, and Drowsiness detection. Image processing is used to identify the driver\'s face, and after that, the image of the driver\'s eyes is extracted to look for indicators of sleepiness. The HAAR face detection algorithm uses image frames that have been captured as input before producing the detected face. To generate results, the model is provided with a sizable database of closed and open eyes. Every time the driver is observed to be sleepy, Buzz alerts the driver. The suggested method for real-time driver drowsiness detection is a practical and affordable approach.

Introduction

I. INTRODUCTION

Driving while fatigued is frequently to blame for fatal car accidents. The majority of drivers are aware of the dangers of texting and driving while intoxicated, but many people underestimate the dangers of sleepy driving. The National Safety Council estimates that drowsiness driving results in approximately 100,000 collisions, 71,000 injuries, and 1,550 fatalities annually (NSC). In research by the AAA Foundation for Traffic Safety, it was discovered that tiredness could potentially be a contributing factor in up to 9.5 percent of collisions overall and 10.8 percent of collisions involving airbag deployment, injuries, or major property damage. Drowsy driving is highly risky, thus it's important for drivers to be aware of the dangers. Drowsy driving must be recognized, and the motorist must be warned [1,5].

Vehicle accidents are frequently caused by drowsy drivers. Lack of sleep has an impact on a person's capacity to perform their daily activities. They react more slowly, have poorer memory and judgement, and their ability to function is compromised. Numerous studies have revealed that inadequate sleep can impair driving just as much as intoxication from alcohol. 40% of those surveyed admitted to having done so at least once while driving, with 20% of those admitting to having dozed off behind the wheel [4].

To sum up, there are a number of issues that need to be resolved:

1. One of the biggest challenges is being able to detect a driver's drowsiness while they are driving and sound an alarm or warning at the right time. Any delay will raise the risk of an accident or failure.
2. The accuracy of the model is occasionally compromised by the low-quality video capturing capabilities of a portable webcam, which prevents it from detecting sleepiness.
3. As a result, there is a demand for more as well as more readily available, flexible technology, like a webcam, that can take better pictures.
4. If the webcam feed occasionally displays a loss of frames or self pressing, which indicates that the output frame is given to the model with a delay of a few seconds and a portion of the alarm sound or warning delay, the likelihood of an accident or failure will rise.

II. LITERATURE REVIEW

In an effort to increase the precision and speed of drowsiness detection, several techniques have been proposed. The development of technologies to reduce the risks of driver drowsiness has been encouraged by traffic-related fatalities and monetary losses. The way the driver's face moves and how quickly their eyes blink are all signs of drowsiness. Electrophysiological signals can also be used to identify drowsy driving.  Smartphones' growing computational capabilities make it possible for computer vision algorithms to function reasonably quickly. The recommended solution employs a smartphone, either an Android or an iPhone, to detect driver fatigue. A smartphone's front camera records images of the driver, which are then sent to the device's CPU for image processing. The face detection and eye detection computer vision algorithms are created using Intel's open-source OpenCV 2.3 computer vision framework for the Android or iOS operating systems. Development of a straightforward, user-friendly, and non-intrusive Android application  that detects drowsiness signs prior to the driver going into the vehicle employing physiological and facial analysis, as well as to identify drowsiness while driving through facial analysis, are the original contributions of the paper [1]. The main goal of is to create a drowsiness detection system for drivers that uses eye and mouth detection to measure yawning. It aims to guarantee the precise detection of yawning expression in the presence of shifting lighting conditions and facial occlusions in order to prevent car accidents. Second, to use a buzzer or beep to notify the driver when drowsiness is detected, and to guarantee a straightforward and effective design that can be executed with hardware and simulation without producing erroneous results. A video camera mounted beneath the front mirror continuously records the driver's face. The first step in detecting the yawn is to identify and monitor the face using the camera's series of frame shots. The mouth area in the identified face is then located, followed by the eyes. When yawning is detected, the closed eye motion is also picked up.  The yawn is then detected using the geometric characteristics of the mouth and eyes [2]. The software will maintain a timer running to record how long the eyes are closed in order to prevent normal blinking from disturbing the drowsiness. The buzzer is sent to alert the driver when the threshold value is exceeded. Continuous eye recording is done with a smartphone camera module so that real-time separation is possible. The counter is also used to track how often the person has become sleepy. The counter limit in our model was set at 15. The owner of the vehicle will be alerted if the counter rises above this threshold so they can alert the driver that it can no longer be safe for them to drive. The work shown in is based on the principles of transfer learning. It made use of the Mobile Net architecture to divide the images. CNN's mobile vision and image segmentation architecture is modelled after Mobile Net [3]. Image processing is used to recognise faces and eyes. A cascade classifier built on HAAR is used for face detection. A moving object tracking algorithm is used to continuously track the eyes. The driver's level of sleepiness was assessed using the PERCLOS algorithm. A camera will be used by the system to monitor the driver's eyes. The output of sound and seat belt vibration is provided when the driver exhibits signs of fatigue. The warning won't be turned off automatically. The ultimate goal is to develop wearable hardware for drowsiness detection, such as smart watches [4].  The driver's face from the recorded video is input and transformed into flowing image frames. To perform image pre-processing, the image frames are kept in a dataset. By using geometric modifications like rotation, translation, and scaling, image pre-processing can boost the quality of image data and enhance some image features, such as face detection features. The LBPH technique is used to extract the features of the mouth and eyes. Using the Haar detection method, the features of the eye (blink) and mouth (yawn) are identified, and parameter measurements like the Eye Aspect Ratio (EAR) and Root Mean Square Error (RMSE) are computed. Check the blink rate of each eye to see if it is open or closed. If the blink rate is higher than the normal threshold, the warning beep system is alerted, and a red signal is displayed. If not, the steps continue to be executed, and a green signal is displayed [5]. Multilayer Perceptron Classifier16, also known as MLP, is used for data processing. MLP is a simple neural network made up of entangled nodes (neurons) that represent the output from the input class. The artificial neuron takes one or more inputs that resemble dendrites, adds them based on connection weights, and then generates a class of outputs [6]. The previously owned calculation converts the image to grayscale by measuring the shading information that is present. The image is divided into smaller regions in order to determine whether or not there is a face there. The application of this equation implies that only the sub regions containing a face are created smoothly. The motion is placed based on the persistent error that is evident when a direct combination of facial development models is used. To differentiate between the location and inclination of the face, an equivalent representation is taken into account. It has a system in place that makes it possible to see changes in the head while also recognising facial motions. On the vehicle driver's device running the Android operating system, the application is installed. The cycle starts when a camera captures live images, which are then sent to a nearby worker. Dlib library is used at the employee's side to recognize facial tourist sites, and a limit regard is used to determine whether or not the driver is sleepy. The EAR (Eye Aspect Ratio) is then processed using these facial milestones, and the results are given back to the driver. In our particular situation, the edge value would be taken as 0.25 and the EAR value obtained at the end of the application would be compared. If the EAR value is not precisely the limit value, this would indicate a weak condition. If drowsiness should occur, a warning would be sent to the driver and other passengers [7].

III. PROPOSED SYSTEM

The proposed strategy aims to reduce the number of daily accidents worldwide and to close the gaps left by previous transport and driver systems. Our system's main objective is to develop a smart system that will be entirely autonomous and will cause the least amount of interference for the driver due to its independence from physical sensors. This is achieved by using a sound alert system to notify the owner of the vehicle and the driver when the latter is discovered to be drowsy. To obtain the most accurate results, the model is trained using a data set containing more than 50000 image samples of open and closed eyes. The model excels at differentiating between closed and open eyes in a variety of lighting situations. The system will keep a timer running to record how long the eyes are closed in order to prevent normal blinking from disturbing the drowsiness. The buzzer is sent to alert the driver when the threshold value is exceeded.

This must be completed in the phases listed below.

1. Phase 1: Face Detection
2. Phase 2: Eyes Detection
3. Phase 3: Drowsiness Detection
4. Phase 4: Alert System

IV. METHOD IMPLEMENTATION

A. System Setup

The suggested driver sleepiness detection system is designed for driving in actual situations. On the dashboard of a car, a webcam can be mounted. The driver's face is the focus of the webcam, which is held horizontally. This camera records video frames, which are gradually processed. The Haar face detection algorithm is used to detect the face in the image frame. The lips and eyes are among the 68 key points used to locate the driver's facial features. Next, both eyes are located in the picture frame. OpenCV is used for real time image processing. The Eye Aspect Ratio (EAR) is employed to ascertain whether the driver's eyes are closed or open. In an effort to prevent natural blinking from upsetting the slumber, the system will continuously monitor the amount of time the eyes are closed. If the threshold value is exceeded, the buzzer alerts the driver.

B.  Algorithm Framework

The Transfer Learning theory serves as the model's foundation. A model created for a task in the past is used as the foundation for a second activity model in the machine learning technique known as transfer learning. Using previously trained models as the foundation for computer vision and natural language processing tasks is a well-liked approach for in-depth learning. Given the substantial computing resources and time required to create neural network models for such issues, transferring learning makes sense. They also yield incredibly precise results for comparable issues. Transfer learning is not the only learning area in deep learning, but it is closely related to problems with multi-tasking and mental retardation. Transfer learning, however, is a well-liked method of in-depth learning when you consider the specialized computer capabilities needed to train in-depth learning models or the sizable and difficult data sets used to train in-depth learning models. Inductive Transfer is the name given to this type of transfer learning used in deep learning.

In order to detect driver drowsiness, the proposed system employs a convolutional neural network (CNN). Deep learning techniques most frequently employ convolutional neural networks (CNNs), a class of artificial neural networks, to analyse visual data. In at least one of their layers, convolutional neural networks (CNNs) take the place of conventional matrix multiplication. They are used in image processing and recognition because they were created specifically to process pixel data. Convolutional layers, pooling layers (maximum, minimum, and average), ReLU layers, and fully connected layers are some of the layers in CNN. The convolution layer consists of kernels (filters), each of which has a width, depth, and height.

C.  HAAR Training

For the detection of faces and features (such as eyes, mouths, sunglasses, etc.), the Open CV library offers functions. These functions can be used to train classifiers. HAAR training is the process of teaching the classifiers to recognise faces. Here, positive and negative images are used to train a cascade function. The sum of the pixels under the various regions of the images is subtracted to create the single value that makes up each feature. A different set of pixels is extracted for each characteristic. The required process will not benefit from all of the extracted features.

D. Face Detection

The suggested approach will begin by individually recording each video frame. Live video processing is supported in great detail by OpenCV. For each frame, the system will find the face in picture. The Haar face detection method is used to do this. The well-known resilient feature-based method Haar cascade is effective in detecting faces in images. The non-face candidates can be eliminated using the Haar algorithm's cascade of phases. Additionally, every phase consists of a mixture of several Haar features, and a Haar feature classifier classifies each feature individually. This system can detect user sleepiness by analysing video stream visuals. Face detection is performed using multiple CNNs kernelized correlation filters, a more sophisticated in-video face tracking method. The driver's face's important features were located using 68 key points, included the lips and eyes. The system measured the ratio of closed eye frames to total frames, the length of continuous eye closure, and the frequency of blinking to assess the driver's level of drowsiness. The DDD system finally alerts the driver.

E.  Eye Detection

The suggested technique makes use of HAAR Cascades to recognise objects in real time, such as the driver's face and eyes. The model makes use of the Open CV library. After identifying the driver's face, the rate at which the eyes blink is used to determine the driver's level of tiredness.

Using the scalar value, Eye Aspect Ratio (EAR) equation may identify an eye blink. For instance, if a driver's eyelids are blinking more frequently, it indicates that they are drowsy. In order to determine the frequency of eye blinking, it is therefore required to precisely identify the form of the eyes. The EAR is utilised as an estimation of the eye opening stage from the landmarks found in the image of the face.

The eye characteristics are identified for each video frame between the calculated height and breadth of the eye. We then set the frame counter "COUNTER" to 0 and the Boolean "ALARM ON" to OFF to keep up with the overall number of frames for both open and closed eyes in addition to the alarm status. We raise the alarm if the recognised eyes are closed for longer than 10 consecutive frames; otherwise, the counter is reset to 0.

F. Drowsiness Detection

It is first necessary to determine whether the eyes are open or closed throughout the time in order to recognise drowsiness. To determine whether the driver's eyes are open or closed, the Eye Aspect Ratio (EAR) is used. After identifying the driver's face, the rate at which the eyes blink is used to determine the driver's level of tiredness. Using the scalar value, Eye Aspect Ratio (EAR) equation may identify an eye blink. For instance, if a driver's eyelids are blinking more frequently, it indicates that they are drowsy. In order to determine the frequency of eye blinking, it is therefore required to precisely identify the form of the eyes. The EAR is utilised as an estimation of the eye opening stage from the landmarks found in the image of the face. The eye characteristics are identified for each video frame between the calculated height and breadth of the eye [8].

The Euclidean formula is used to determine the EAR by measuring the distance between the important points on the eyes. When the eye is open, the aspect ratio will be greater and more stable over time. When a blink occurs, the eye aspect ratio will decrease to almost zero.

While the denominator of this equation calculates the distance between horizontal eye landmarks, the numerator of the equation computes the distance between vertical eye landmarks. Following the acquisition of eyeballs, the algorithm calculates the total number for open eyes in each frame to ascertain tiredness. The motorist is considered to be sleepy if the conditions are met. The system's associated buzzer makes corrections to the driver's aberrant behaviour.

V. RESULT

The model is remarkably accurate at differentiating between closed and open eyes. Our system uses a webcam to capture images, which are then processed using OpenCV image processing libraries, which is free and open-source software. The alarm is then set off after the Haar cascade algorithm recognises the person's face and eye.

The tests were carried out under a variety of circumstances, including:

1. Various lighting conditions
2. Drivers wearing spectacles

   VI. FUTURE SCOPE

1. The current model only considers whether or not someone's eyes are closed when determining whether they are sleepy. However, yawning and nodding detection can help the model even more. Although yawning or nodding alone doesn't produce accurate and reliable results. However, when combined with eye monitoring, it could produce much more precise results.
2. Infrared cameras might be employed to enhance performance in dim lighting.
3. More data can be added to the model to further refine it. The model will produce more accurate and consistent results the more data that is fed into it. There is room for improvement in the model, which has been trained for about 50000 images.

Conclusion

To prevent drowsy driving-related accidents, Driver Drowsiness Detection was developed to assist drivers in staying awake while operating a motor vehicle. This study focused on drowsy drivers and their propensity to cause auto accidents. The driver drowsiness detection system calculates the driver\'s tiredness level from the driver using a combination of webcam, HAAR cascade classifier, drowsiness detection is used to calculate whether or not a driver is drowsy. It simultaneously retrieves photos from the camera, which is sufficient to identify a driver\'s features in real time. The system processes the captured images using open-source software called Open CV image processing libraries. The system becomes a low-cost drowsiness detection system when a webcam and an Open CV are combined.

References

[1] Mali, S., Bangale, K., Walekar, R., & Shitole, P. T. Driver Drowsiness Detection Using Smartphone. [2] Piyush, R. K., & Dahiya, N. (2019). Driver Distraction and Drowsiness Detection System. [3] Khetan, D., Nawani, A., Aggarwal, A., & Kaur, M. S. Driver Drowsiness Detection in Real-time. [4] Kiran, V. N., Raksha, R., Anisoor, R., Varsha, K. N., & Nagamani, N. P. (2020). Driver drowsiness detection. Int J Eng Res Technol (IJERT), 8(15), 33-35. [5] Manjunath, C. R., Krishna, K. S., Sree, K. B., & Ramyatha, K. N. (2022, May). Driver Drowsiness Detection Alert System Using Haar Method. In International Conference on Intelligent Emerging Methods of Artificial Intelligence & Cloud Computing: Proceedings of IEMAICLOUD 2021 (pp. 380-389). Cham: Springer International Publishing. [6] Jabbar, R., Al-Khalifa, K., Kharbeche, M., Alhajyaseen, W., Jafari, M., & Jiang, S. (2018). Real-time driver drowsiness detection for android application using deep neural networks techniques. Procedia computer science, 130, 400-407. [7] Sajikumar, A., Aji, A., Baby, J., Prasad, A., Stanly, A. K., & John, N. M. (2021). Save the drowsy driver drowsy driver detection. [8] Musale, T., & Pansambal, B. H. (2017). Driver Drowsiness Detection Technique Using Raspberry pi. International Journal of Development Research, 7(02), 49-53.

Copyright

Copyright © 2023 Sanjeevani Srivastava, Swekcha Srivastava, Shivangi Srivastava, Sakshi Mishra. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.